Augmented Reality in der Kopf-Hals-Chirurgie

Claudia Scherl; Sonja Ludwig; Jürgen Hesser; Sara MonjiAzad; Jan Stallkamp; Frederic Jungbauer; Frederik Enders; Cleo-Aron Weis; Nicole Rotter

doi:10.1055/a-2463-4599

Laryngo-Rhino-Otologie, Table of Contents

Laryngorhinootologie
DOI: 10.1055/a-2463-4599

Übersicht

Augmented Reality in der Kopf-Hals-Chirurgie

Augmented Reality in head and neck surgery

Claudia Scherl

¹Klinik für Hals-Nasen-Ohrenheilkunde, Kopf- und Halschirurgie, Ruprecht-Karls-Universität Heidelberg, Medizinische Fakultät Mannheim, Mannheim, Germany

²AI Health Innovation Cluster, Heidelberg-Mannheim Health and Life Science Alliance, Ruprecht-Karls-Universität Heidelberg, Heidelberg, Germany (Ringgold ID: RIN9144)

,

Sonja Ludwig

¹Klinik für Hals-Nasen-Ohrenheilkunde, Kopf- und Halschirurgie, Ruprecht-Karls-Universität Heidelberg, Medizinische Fakultät Mannheim, Mannheim, Germany

,

Jürgen Hesser

²AI Health Innovation Cluster, Heidelberg-Mannheim Health and Life Science Alliance, Ruprecht-Karls-Universität Heidelberg, Heidelberg, Germany (Ringgold ID: RIN9144)

³Mannheim Institut for Intelligent Systems in Medicine (MIISM), Ruprecht-Karls-Universität Heidelberg, Mannheim, Germany (Ringgold ID: RIN9144)

⁴Interdisciplinary Center for Scientific Computing (IWR), Ruprecht-Karls-Universität Heidelberg, Heidelberg, Germany (Ringgold ID: RIN9144)

⁵Zentrales Institut für Technische Informatik, Ruprecht-Karls-Universität Heidelberg, Heidelberg, Germany (Ringgold ID: RIN9144)

,

Sara MonjiAzad

³Mannheim Institut for Intelligent Systems in Medicine (MIISM), Ruprecht-Karls-Universität Heidelberg, Mannheim, Germany (Ringgold ID: RIN9144)

,

Jan Stallkamp

³Mannheim Institut for Intelligent Systems in Medicine (MIISM), Ruprecht-Karls-Universität Heidelberg, Mannheim, Germany (Ringgold ID: RIN9144)

,

Frederic Jungbauer

¹Klinik für Hals-Nasen-Ohrenheilkunde, Kopf- und Halschirurgie, Ruprecht-Karls-Universität Heidelberg, Medizinische Fakultät Mannheim, Mannheim, Germany

,

Frederik Enders

⁶Neurochirurgische Klinik, Ruprecht-Karls-Universität Heidelberg, Medizinische Fakultät Mannheim, Mannheim, Germany

,

Cleo-Aron Weis

⁷Pathologisches Institut, Ruprecht-Karls-Universität Heidelberg, Heidelberg, Germany (Ringgold ID: RIN9144)

,

Nicole Rotter

¹Klinik für Hals-Nasen-Ohrenheilkunde, Kopf- und Halschirurgie, Ruprecht-Karls-Universität Heidelberg, Medizinische Fakultät Mannheim, Mannheim, Germany

› Author Affiliations

Abstract

Zusammenfassung

Augmented Reality (AR) ist eine aufstrebende Technologie, die Operateure beim räumlichen Denken während klinischer Eingriffe unterstützt. Zu den faszinierendsten Anwendungen gehören Visualisierungen von anatomischen Strukturen, Schnittbildern oder Zugangswegen, die in Kombination mit der realen Umgebung direkt am Patienten angezeigt werden können. Das Ziel dieser narrativen Übersicht besteht darin, den Stand der Technik und die zukünftigen Richtungen der AR in der Kopf-Hals-Chirurgie zu skizzieren. Besonderes Augenmerk liegt dabei auf der Darlegung des Potenzials und der noch zu lösenden Schwierigkeiten dieser neuen Technik.

Abstract

Augmented Reality (AR) is an emerging technology that supports surgeons in spatial reasoning during clinical procedures. The most fascinating applications include visualizations of anatomical structures, cross-sectional images or surgical approaches, which can be displayed directly on the patient in combination with the real world. The aim of this narrative review is to outline the state of the art and future directions of AR in head and neck surgery. Particular attention is paid to explaining the potential and the difficulties that still need to be solved with this new technology.

Schlüsselwörter

Augmented Reality - Image Guidance - Navigation - virtuell - dreidimensional

Keywords

Augmented Reality - Image Guidance - Navigation - virtual - three dimensional

Full Text

References

Literatur
1 Ayoub A, Pulijala Y. The application of virtual reality and augmented reality in Oral & Maxillofacial Surgery. BMC Oral Health 2019; 19: 238
2 Heilig ML. Sensorama simulator. US PAT 3,050,870. 1962
3 Citardi MJ, Agbetoba A, Bigcas JL. et al. Augmented reality for endoscopic sinus surgery with surgical navigation: a cadaver study. International forum of allergy & rhinology 2016; 523-528
4 Ramakrishnan VR, Orlandi RR, Citardi MJ. et al. The use of image-guided surgery in endoscopic sinus surgery: an evidence-based review with recommendations. International forum of allergy & rhinology 2013; 236-241
5 Chand M, Keller DS, Devoto L. et al. Furthering precision in sentinel node navigational surgery for Oral Cancer: a novel triple targeting system. Journal of Fluorescence 2018; 28: 483-486
6 Marmulla R, Hoppe H, Mühling J. et al. New augmented reality concepts for craniofacial surgical procedures. Plastic and Reconstructive Surgery 2005; 115: 1124-1128
7 Li L, Yang J, Chu Y. et al. A novel augmented reality navigation system for endoscopic sinus and skull base surgery: a feasibility study. PLoS One 2016; 11: e0146996
8 Suzuki N, Hattori A, Iimura J. et al. Development of AR Surgical Navigation Systems for Multiple Surgical Regions. MMVR 2014; 404-408
9 Salisbury JP, Keshav NU, Sossong AD. et al. Concussion assessment with smartglasses: validation study of balance measurement toward a lightweight, multimodal, field-ready platform. JMIR mHealth and uHealth 2018; 6: e8478
10 Scherl C, Männle D, Rotter N. et al. Augmented reality during parotid surgery: real-life evaluation of voice control of a head mounted display. Eur Arch Otorhinolaryngol 2023; 280: 2043-2049
11 Sahovaler A, Chan HH, Gualtieri T. et al. Augmented reality and intraoperative navigation in sinonasal malignancies: A preclinical study. Frontiers in Oncology 2021; 11: 723509
12 Khamene A, Acker FW, Vogt S. et al. An augmented reality system for MRI-guided needle biopsies. Medicine Meets Virtual Reality 11. IOS Press; 2003. 11.
13 McJunkin JL, Jiramongkolchai P, Chung W. et al. Development of a mixed reality platform for lateral skull base anatomy. Otology & Neurotology: Official Publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology 2018; 39: e1137
14 Meola A, Cutolo F, Carbone M. et al. Augmented reality in neurosurgery: a systematic review. Neurosurgical review 2017; 40: 537-548
15 Scherl C, Stratemeier J, Karle C. et al. Augmented reality with HoloLens in parotid surgery: how to assess and to improve accuracy. European Archives of Oto-Rhino-Laryngology 2021; 278: 2473-2483
16 Wong K, Yee HM, Xavier BA. et al. Applications of augmented reality in otolaryngology: a systematic review. Otolaryngology–Head and Neck Surgery 2018; 159: 956-967
17 Chen JX, Yu SE, Ding AS. et al. Augmented reality in otology/neurotology: a scoping review with implications for practice and education. The Laryngoscope 2023; 133: 1786-1795
18 Colombo E, Fick T, Esposito G. et al. Segmentation techniques of brain arteriovenous malformations for 3D visualization: a systematic review. Radiol Med 2022; 127: 1333-1341
19 Ke J, Lv Y, Ma F. et al. Deep learning-based approach for the automatic segmentation of adult and pediatric temporal bone computed tomography images. Quant Imaging Med Surg 2023; 13: 1577-1591
20 Andrews CM, Henry AB, Soriano IM. et al. Registration Techniques for Clinical Applications of Three-Dimensional Augmented Reality Devices. IEEE J Transl Eng Health Med 2021; 9: 4900214
21 McJunkin JL, Jiramongkolchai P, Chung W. et al. Development of a Mixed Reality Platform for Lateral Skull Base Anatomy. Otol Neurotol 2018; 39: e1137-e1142
22 Li Y, Chen X, Wang N. et al. A wearable mixed-reality holographic computer for guiding external ventricular drain insertion at the bedside. J Neurosurg 2018; 1-8
23 Frantz T, Jansen B, Duerinck J. et al. Augmenting Microsoft's HoloLens with vuforia tracking for neuronavigation. Healthc Technol Lett 2018; 5: 221-225
24 Condino S, Sannino S, Cutolo F. et al. Single feature constrained manual registration method for Augmented Reality applications in gynecological laparoscopic interventions. Annu Int Conf IEEE Eng Med Biol Soc 2022; 2022: 566-571
25 Morita S, Suzuki K, Yamamoto T. et al. Out-of-Plane Needle Placements Using 3D Augmented Reality Protractor on Smartphone: An Experimental Phantom Study. Cardiovasc Intervent Radiol 2023; 46: 675-679
26 Jiang T, Yu D, Wang Y. et al. HoloLens-Based Vascular Localization System: Precision Evaluation Study With a Three-Dimensional Printed Model. J Med Internet Res 2020; 22: e16852
27 Müller F, Roner S, Liebmann F. et al. Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J 2020; 20: 621-628
28 Meulstee JW, Nijsink J, Schreurs R. et al. Toward Holographic-Guided Surgery. Surg Innov 2019; 26: 86-94
29 Hogan J. Augmedics’ xvision Spine system. Accessed May 21, 2024 at: https://www.accessdata.fda.gov/cdrh_docs/pdf19/K190929.pdf
30 Chien JC, Tsai Y-R, Wu CT. et al. HoloLens-based AR system with a robust point set registration algorithm. Sensors 2019; 19: 3555
31 Gsaxner C, Wallner J, Chen X. et al. Facial model collection for medical augmented reality in oncologic cranio-maxillofacial surgery. Scientific data 2019; 6: 310
32 Kwon JG, Hong DW, Choi JW. Clinical Applications of Augmented Reality Technology in Microsurgical Planning of Head and Neck Reconstruction. J Craniofac Surg 2022; 33: 863-866
33 Caversaccio M, Langlotz F, Nolte LP. et al. Impact of a self-developed planning and self-constructed navigation system on skull base surgery: 10 years experience. Acta Otolaryngol 2007; 127: 403-407
34 Barber SR, Wong K, Kanumuri V. et al. Augmented reality, surgical navigation, and 3D printing for transcanal endoscopic approach to the petrous apex. OTO open 2018; 2: 2473974X18804492
35 Tel A, Zeppieri M, Robiony M. et al. Exploring Deep Cervical Compartments in Head and Neck Surgical Oncology through Augmented Reality Vision: A Proof of Concept. J Clin Med 2023; 12
36 Goto Y, Kawaguchi A, Inoue Y. et al. Efficacy of a Novel Augmented Reality Navigation System Using 3D Computer Graphic Modeling in Endoscopic Transsphenoidal Surgery for Sellar and Parasellar Tumors. Cancers (Basel) 2023; 15
37 Bopp MHA, Saß B, Pojskić M. et al. Use of Neuronavigation and Augmented Reality in Transsphenoidal Pituitary Adenoma Surgery. J Clin Med 2022; 11
38 Dixon BJ, Daly MJ, Chan H. et al. Augmented real-time navigation with critical structure proximity alerts for endoscopic skull base surgery. The Laryngoscope 2014; 124: 853-859
39 Linxweiler M, Pillong L, Kopanja D. et al. Augmented reality-enhanced navigation in endoscopic sinus surgery: A prospective, randomized, controlled clinical trial. Laryngoscope Investigative Otolaryngology 2020; 5: 621-629
40 Necker FN, Chang M, Leuze C. et al. Virtual Resection Specimen Interaction Using Augmented Reality Holograms to Guide Margin Communication and Flap Sizing. Otolaryngology – Head and Neck Surgery 2023;
41 Pratt P, Ives M, Lawton G. et al. Through the HoloLens™ looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels. European radiology experimental 2018; 2: 1-7
42 Gorpas D, Phipps J, Bec J. et al. Autofluorescence lifetime augmented reality as a means for real-time robotic surgery guidance in human patients. Scientific reports 2019; 9: 1187
43 Liu WP, Richmon JD, Sorger JM. et al. Augmented reality and cone beam CT guidance for transoral robotic surgery. Journal of robotic surgery 2015; 9: 223-233
44 Agbetoba A, Luong A, Siow JK. et al. Educational utility of advanced three-dimensional virtual imaging in evaluating the anatomical configuration of the frontal recess. International forum of allergy & rhinology 2017; 07 (02) 143-148
45 Küçük S, Kapakin S, Göktaş Y. Learning anatomy via mobile augmented reality: Effects on achievement and cognitive load. Anat Sci Educ 2016; 9: 411-421
46 Bong JH, Song Hj, Oh Y. et al. Endoscopic navigation system with extended field of view using augmented reality technology. The International Journal of Medical Robotics and Computer Assisted Surgery 2018; 14: e1886
47 Chen X, Xu L, Wang Y. et al. Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display. Journal of biomedical informatics 2015; 55: 124-131
48 Pollefeys M. Microsoft HoloLens facilitates computer vision research by providing access to raw image sensor streams with Research Mode. Accessed May 21, 2024 at: https://www.microsoft.com/en-us/research/blog/microsoft-hololens-facilitates-computer-vision-research-by-providing-access-to-raw-image-sensor-streams-with-research-mode/
49 de Oliveira ME, Debarba HG, Lädermann A. et al. A hand-eye calibration method for augmented reality applied to computer-assisted orthopedic surgery. Int J Med Robot 2019; 15: e1969
50 Ahmed OH, Marcus S, Lebowitz RA. et al. Evolution in visualization for sinus and skull base surgery: from headlight to endoscope. Otolaryngologic Clinics of North America 2017; 50: 505-519
51 Tzelnick S, Rampinelli V, Sahovaler A. et al. Skull-Base Surgery – A Narrative Review on Current Approaches and Future Developments in Surgical Navigation. Journal of Clinical Medicine 2023; 12: 2706
52 Marroquin R, Lalande A, Hussain R. et al. Augmented reality of the middle ear combining otoendoscopy and temporal bone computed tomography. Otology & Neurotology 2018; 39: 931-939
53 Creighton FX, Unberath M, Song T. et al. Early feasibility studies of augmented reality navigation for lateral skull base surgery. Otology & Neurotology 2020; 41: 883-888
54 Liu WP, Azizian M, Sorger J. et al. Cadaveric feasibility study of da vinci si–assisted cochlear implant with augmented visual navigation for otologic surgery. JAMA Otolaryngology – Head & Neck Surgery 2014; 140: 208-214
55 Szewczyk M, Golusinski W, Pazdrowski J. et al. Positive fresh frozen section margins as an adverse independent prognostic factor for local recurrence in oral cancer patients. The Laryngoscope 2018; 128: 1093-1098
56 Kreeft AM, Smeele LE, Rasch CR. et al. Preoperative imaging and surgical margins in maxillectomy patients. Head & neck 2012; 34: 1652-1656
57 Chan HH, Sahovaler A, Daly MJ. et al. Projected cutting guides using an augmented reality system to improve surgical margins in maxillectomies: A preclinical study. Oral Oncology 2022; 127: 105775
58 Seetohul J, Shafiee M, Sirlantzis K. Augmented Reality (AR) for Surgical Robotic and Autonomous Systems: State of the Art, Challenges, and Solutions. Sensors (Basel) 2023; 23
59 Iqbal H, Tatti F, Rodriguez YBF. Augmented reality in robotic assisted orthopaedic surgery: A pilot study. J Biomed Inform 2021; 120: 103841
60 Dedhia RC, Lord CA, Pinheiro-Neto CD. et al. Endoscopic endonasal pituitary surgery: impact of surgical education on operation length and patient morbidity. Journal of Neurological Surgery Part B: Skull Base 2012; 405-409
61 Dombrowski T, Dazert S, Volkenstein S. Strategies of Digitized Learning. Laryngorhinootologie 2019; 98: S197-s219
62 Offergeld C, von Schnakenburg P, Knopf A. et al. Innovation trifft Tradition: Virtual Reality (VR) in der HNO-Lehre. Laryngo-Rhino-Otologie 2022; 101: 295-297
63 Botden SM, Buzink SN, Schijven MP. et al. Augmented versus virtual reality laparoscopic simulation: what is the difference? A comparison of the ProMIS augmented reality laparoscopic simulator versus LapSim virtual reality laparoscopic simulator. World journal of surgery 2007; 31: 764-772
64 James J, Irace AL, Gudis DA. et al. Simulation training in endoscopic skull base surgery: A scoping review. World Journal of Otorhinolaryngology – Head and Neck Surgery 2022; 8: 73-81
65 Kovoor JG, Gupta AK, Gladman MA. Validity and effectiveness of augmented reality in surgical education: a systematic review. Surgery 2021; 170: 88-98
66 Yong M, Pauwels J, Kozak FK. et al. Application of augmented reality to surgical practice: A pilot study using the ODG R7 Smartglasses. Clinical Otolaryngology 2020; 45: 130-134
67 Vera AM, Russo M, Mohsin A. et al. Augmented reality telementoring (ART) platform: a randomized controlled trial to assess the efficacy of a new surgical education technology. Surg Endosc 2014; 28: 3467-3472
68 El-Sabawi B, Magee III W. The evolution of surgical telementoring: current applications and future directions. Annals of translational medicine 2016; 4
69 Baumhauer M, Feuerstein M, Meinzer HP. et al. Navigation in endoscopic soft tissue surgery: perspectives and limitations. Journal of endourology 2008; 22: 751-766
70 Mangano FG, Admakin O, Lerner H. et al. Artificial intelligence and augmented reality for guided implant surgery planning: A proof of concept. J Dent 2023; 133: 104485
71 Chan HH, Haerle SK, Daly MJ. et al. An integrated augmented reality surgical navigation platform using multi-modality imaging for guidance. PLoS One 2021; 16: e0250558
72 Tabrizi LB, Mahvash M. Augmented reality–guided neurosurgery: accuracy and intraoperative application of an image projection technique. Journal of neurosurgery 2015; 123: 206-211
73 Luzon JA, Stimec BV, Bakka AO. et al. Value of the surgeon’s sightline on hologram registration and targeting in mixed reality. International journal of computer assisted radiology and surgery 2020; 15: 2027-2039
74 Lee S, Hu X, Hua H. Effects of optical combiner and IPD change for convergence on near-field depth perception in an optical see-through HMD. IEEE transactions on visualization and computer graphics 2015; 22: 1540-1554
75 Lai M, Skyrman S, Shan C. et al. Fusion of augmented reality imaging with the endoscopic view for endonasal skull base surgery; a novel application for surgical navigation based on intraoperative cone beam computed tomography and optical tracking. Plos one 2020; 15: e0227312
76 Chu Y, Yang J, Ma S. et al. Registration and fusion quantification of augmented reality based nasal endoscopic surgery. Medical image analysis 2017; 42: 241-256
77 Ruggiero F, Cercenelli L, Emiliani N. et al. Preclinical Application of Augmented Reality in Pediatric Craniofacial Surgery: An Accuracy Study. Journal of Clinical Medicine 2023; 12: 2693
78 Jiang T, Zhu M, Chai G. et al. Precision of a novel craniofacial surgical navigation system based on augmented reality using an occlusal splint as a registration strategy. Scientific reports 2019; 9: 501
79 Prasad K, Miller A, Sharif K. et al. Augmented-reality surgery to guide head and neck cancer re-resection: a feasibility and accuracy study. Annals of surgical oncology 2023; 1-7
80 Dixon BJ, Daly MJ, Chan HH. et al. Inattentional blindness increased with augmented reality surgical navigation. American journal of rhinology & allergy 2014; 28: 433-437